<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.9.1"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>AIfES 2: Tutorial inference Q7</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
</script>
<script type="text/javascript" async="async" src="https://cdn.jsdelivr.net/npm/mathjax@2/MathJax.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="AIfES_logo_small.png"/></td>
  <td id="projectalign" style="padding-left: 0.5em;">
   <div id="projectname">AIfES 2
   &#160;<span id="projectnumber">2.0.0</span>
   </div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.9.1 -->
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
var searchBox = new SearchBox("searchBox", "search",false,'Search','.html');
/* @license-end */
</script>
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
$(function() {
  initMenu('',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
/* @license-end */</script>
<div id="main-nav"></div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
  <div id="nav-tree">
    <div id="nav-tree-contents">
      <div id="nav-sync" class="sync"></div>
    </div>
  </div>
  <div id="splitbar" style="-moz-user-select:none;" 
       class="ui-resizable-handle">
  </div>
</div>
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
$(document).ready(function(){initNavTree('_tutorial_inference_q7.html',''); initResizable(); });
/* @license-end */
</script>
<div id="doc-content">
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div class="PageDoc"><div class="header">
  <div class="headertitle">
<div class="title">Tutorial inference Q7 </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p><a class="anchor" id="md__builds_ims_esa_aifes_2_doc_md_05_tutorial_inference_q7"></a> This tutorial shows based on an example, how to perform an inference in AIfES on an integer quantized Feed-Forward Neural Network (FNN). It is assumed, that the trained weights are already available or will be calculated with external tools on a PC. If you want to train the neural network with AIfES, switch to the <a class="el" href="_tutorial_training_f32.html">training tutorial</a>.</p>
<h1><a class="anchor" id="autotoc_md31"></a>
Example</h1>
<p>As an example, we take a robot with two powered wheels and a RGB color sensor that should follow a black line on a white paper. To fulfill the task, we map the color sensor values with an FNN directly to the control commands for the two wheel-motors of the robot. The inputs for the FNN are the RGB color values scaled to the interval [0, 1] and the outputs are either "on" (1) or "off" (0) to control the motors.</p>
<div class="image">
<img src="example_line_follow_robot.png" alt="" width="600px"/>
</div>
<p>The following cases should be considered:</p><ol type="1">
<li>The sensor points to a black area (RGB = [0, 0, 0]): The robot is too far on the left and should turn on the left wheel-motor while removing power from the right motor.</li>
<li>The sensor points to a white area (RGB = [1, 1, 1]): The robot is too far on the right and should turn on the right wheel-motor while removing power from the left motor.</li>
<li>The sensor points to a red area (RGB = [1, 0, 0]): The robot reached the stop mark and should switch off both motors.</li>
</ol>
<p>The resulting input data of the FNN is then</p>
<table class="markdownTable">
<tr class="markdownTableHead">
<th class="markdownTableHeadNone">R   </th><th class="markdownTableHeadNone">G   </th><th class="markdownTableHeadNone">B    </th></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">0    </td></tr>
<tr class="markdownTableRowEven">
<td class="markdownTableBodyNone">1   </td><td class="markdownTableBodyNone">1   </td><td class="markdownTableBodyNone">1    </td></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">1   </td><td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">0   </td></tr>
</table>
<p>and the output should be</p>
<table class="markdownTable">
<tr class="markdownTableHead">
<th class="markdownTableHeadNone">left motor   </th><th class="markdownTableHeadNone">right motor    </th></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">1   </td><td class="markdownTableBodyNone">0    </td></tr>
<tr class="markdownTableRowEven">
<td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">1    </td></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">0   </td></tr>
</table>
<h1><a class="anchor" id="autotoc_md32"></a>
Design the neural network</h1>
<p>To set up the FNN in AIfES 2, we need to design the structure of the neural network. It needs three inputs for the RGB color values and two outputs for the two motors. Because the task is rather easy, we use just one hidden layer with three neurons.</p>
<div class="image">
<img src="example_line_follow_abstract_model.png" alt="" width="800px"/>
</div>
<p>To create the network in AIfES, it must be divided into logical layers like the fully-connected (dense) layer and the activation functions. We choose a Leaky ReLU activation for the hidden layer and a Sigmoid activation for the output layer.</p>
<div class="image">
<img src="example_line_follow_model.png" alt="" width="800px"/>
</div>
<h1><a class="anchor" id="GetWeightsAndBiases"></a>
Get the pre-trained weights and biases</h1>
<p>To perform an inference you need the trained weights and biases of the model. For example you can train your model with Keras or PyTorch, extract the weights and biases and copy them to your AIfES model. <br  />
 For a dense layer, AIfES expects the weights as a matrix of shape [Inputs x Outputs] and the bias as a matrix of shape [1 x Outputs].</p>
<p>Example model in Keras: </p><div class="fragment"><div class="line">model = Sequential()</div>
<div class="line"> </div>
<div class="line">model.add(Input(shape=(3,)))</div>
<div class="line">model.add(Dense(3))</div>
<div class="line">model.add(LeakyReLU(alpha=0.01))</div>
<div class="line">model.add(Dense(2))</div>
<div class="line">model.add(Activation(<span class="stringliteral">&#39;sigmoid&#39;</span>))</div>
</div><!-- fragment --><p>Example model in PyTorch: </p><div class="fragment"><div class="line"><span class="keyword">class </span>Net(nn.Module):</div>
<div class="line"> </div>
<div class="line">    <span class="keyword">def </span>__init__(self):</div>
<div class="line">        super(Net, self).__init__()</div>
<div class="line">        </div>
<div class="line">        self.dense_layer_1 = nn.Linear(3, 3)</div>
<div class="line">        self.leaky_relu_layer = nn.LeakyReLU(0.01)</div>
<div class="line">        self.dense_layer_2 = nn.Linear(3, 2)</div>
<div class="line">        self.sigmoid_layer = nn.Sigmoid()</div>
<div class="line"> </div>
<div class="line">    <span class="keyword">def </span>forward(self, x):</div>
<div class="line">        x = self.dense_layer_1(x)</div>
<div class="line">        x = self.leaky_relu_layer(x)</div>
<div class="line">        x = self.dense_layer_2(x)</div>
<div class="line">        x = self.sigmoid_layer(x)</div>
<div class="line">        <span class="keywordflow">return</span> x</div>
</div><!-- fragment --><p>Our example weights and biases for the two dense layers after training are: </p><p class="formulaDsp">
\[ w_1 = \left( \begin{array}{c} 3.64540 &amp; -3.60981 &amp; 1.57631 \\ -2.98952 &amp; -1.91465 &amp; 3.06150 \\ -2.76578 &amp; -1.24335 &amp; 0.71257 \end{array}\right) \]
</p>
 <p class="formulaDsp">
\[ b_1 = \left( \begin{array}{c} 0.72655 &amp; 2.67281 &amp; -0.21291 \end{array}\right) \]
</p>
 <p class="formulaDsp">
\[ w_2 = \left( \begin{array}{c} -1.09249 &amp; -2.44526 \\ 3.23528 &amp; -2.88023 \\ -2.51201 &amp; 2.52683 \end{array}\right) \]
</p>
 <p class="formulaDsp">
\[ b_2 = \left( \begin{array}{c} 0.14391 &amp; -1.34459 \end{array}\right) \]
</p>
<h1><a class="anchor" id="autotoc_md33"></a>
Q7 quantization</h1>
<h2><a class="anchor" id="autotoc_md34"></a>
General</h2>
<p>The Q7 quantization is an asymmetric 8 bit integer quantization that allows integer-only calculations on real values. The quantization procedure is closely related to the proposed techniques of the paper <a href="https://arxiv.org/abs/1712.05877">"Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference"</a>. A real value \( r \) is represented by an integer value \( q \), the scaling factor / shift \( s \) and the zero point \( z \) according to the following formula: </p><p class="formulaDsp">
\[ r = 2^{-s} * (q - z) \]
</p>
<p>To get the quantized value \( q \) out of a real value \( r \) you have to calculate </p><p class="formulaDsp">
\[ q = round(\frac{r}{2^{-s}} + z) \]
</p>
<p>To store the zero point and the scaling factor, a Q7 tensor needs additional parameters. These are stored in a structure of type <code>aimath_q7_params_t</code> and set to the <code>tensor_params</code> field of an aitensor_t. For example the tensor</p>
<p class="formulaDsp">
\[ \left( \begin{array}{c} 0 &amp; 1 &amp; 2 \\ 3 &amp; 4 &amp; 5 \end{array}\right) \]
</p>
<p> can be created with </p><div class="fragment"><div class="line">int8_t example_data[] = {0, 2, 4</div>
<div class="line">                         6, 8, 10};</div>
<div class="line">uint16_t example_shape[] = {2, 3};</div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> example_q_params = { .<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 1, .zero_point = 0 };</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> example_tensor = {</div>
<div class="line">    .<a class="code" href="structaitensor.html#abf7a461af7296a09d5246ca13591b988">dtype</a> = <a class="code" href="aimath__q7_8h.html#a41e69a98d91e98757ac05f122fd8326f">aiq7</a>,</div>
<div class="line">    .dim = 2,</div>
<div class="line">    .shape = example_shape,</div>
<div class="line">    .tensor_params = example_q_params,</div>
<div class="line">    .data = example_data</div>
<div class="line">};</div>
<div class="ttc" id="aaimath__q7_8h_html_a41e69a98d91e98757ac05f122fd8326f"><div class="ttname"><a href="aimath__q7_8h.html#a41e69a98d91e98757ac05f122fd8326f">aiq7</a></div><div class="ttdeci">const aimath_dtype_t * aiq7</div><div class="ttdoc">The Q7 data-type indicator.</div></div>
<div class="ttc" id="astructaimath__q7__params_html"><div class="ttname"><a href="structaimath__q7__params.html">aimath_q7_params</a></div><div class="ttdoc">Parameters used for the quantized Q7  values, used as property of a tensor.</div><div class="ttdef"><b>Definition:</b> aimath_q7.h:148</div></div>
<div class="ttc" id="astructaimath__q7__params_html_a8f20d77488fe485aaf9ea8e9c5d7d35e"><div class="ttname"><a href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">aimath_q7_params::shift</a></div><div class="ttdeci">uint16_t shift</div><div class="ttdoc">The scaling factor  of the quantization (The total scale is calculated with )</div><div class="ttdef"><b>Definition:</b> aimath_q7.h:149</div></div>
<div class="ttc" id="astructaitensor_html"><div class="ttname"><a href="structaitensor.html">aitensor</a></div><div class="ttdoc">A tensor in AIfES.</div><div class="ttdef"><b>Definition:</b> aifes_math.h:89</div></div>
<div class="ttc" id="astructaitensor_html_abf7a461af7296a09d5246ca13591b988"><div class="ttname"><a href="structaitensor.html#abf7a461af7296a09d5246ca13591b988">aitensor::dtype</a></div><div class="ttdeci">const aimath_dtype_t * dtype</div><div class="ttdoc">The datatype of the tensor, e.g.</div><div class="ttdef"><b>Definition:</b> aifes_math.h:90</div></div>
</div><!-- fragment --><p> or </p><div class="fragment"><div class="line">int8_t example_data[] = {0, 2, 4</div>
<div class="line">                         6, 8, 10};</div>
<div class="line">uint16_t example_shape[] = {2, 3};</div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> example_q_params = { .<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 1, .zero_point = 0 };</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> example_tensor = AITENSOR_2D_Q7(example_shape, &amp;example_q_params, example_data);</div>
</div><!-- fragment --><p>Scalar values can be created with the structure <code>aiscalar_q7_t</code> and initialized either automatically with </p><div class="fragment"><div class="line"><a class="code" href="structaiscalar__q7.html">aiscalar_q7_t</a> scalar = AISCALAR_Q7(4.2f, 4, 0);</div>
<div class="ttc" id="astructaiscalar__q7_html"><div class="ttname"><a href="structaiscalar__q7.html">aiscalar_q7</a></div><div class="ttdoc">Single quantized Q7  value/scalar.</div><div class="ttdef"><b>Definition:</b> aimath_q7.h:155</div></div>
</div><!-- fragment --><p> or manually with </p><div class="fragment"><div class="line"><a class="code" href="structaiscalar__q7.html">aiscalar_q7_t</a> scalar = { .<a class="code" href="structaiscalar__q7.html#af6cd29ae1beb8f0d3e03c9824b5ffa63">value</a> = 67, .shift = 4, .zero_point = 0};</div>
<div class="ttc" id="astructaiscalar__q7_html_af6cd29ae1beb8f0d3e03c9824b5ffa63"><div class="ttname"><a href="structaiscalar__q7.html#af6cd29ae1beb8f0d3e03c9824b5ffa63">aiscalar_q7::value</a></div><div class="ttdeci">int8_t value</div><div class="ttdoc">Quantized value .</div><div class="ttdef"><b>Definition:</b> aimath_q7.h:156</div></div>
</div><!-- fragment --><p>Also the macro functions <code>FLOAT_TO_Q7(float_value, shift, zero_point)</code> and <code>Q7_TO_FLOAT(integer_value, shift, zero_point)</code> are available for quick and easy conversion between float and integer values.</p>
<h2><a class="anchor" id="autotoc_md35"></a>
Tensor quantization helper</h2>
<p>AIfES provides some helper functions to quantize 32 bit float values to 8 bit integer values. The following example shows how to automatically quantize a tensor with AIfES:</p>
<p>Asymmetric quantization (zero point != 0): </p><div class="fragment"><div class="line"><span class="comment">// Source tensor with 32 bit float values</span></div>
<div class="line"><span class="keywordtype">float</span> example_data_f32[] = {0.0f, 1.0f, 2.0f,</div>
<div class="line">                            3.0f, 4.0f, 5.0f};</div>
<div class="line">uint16_t example_shape_f32[] = {2, 3};</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> example_tensor_f32 = AITENSOR_2D_F32(example_shape_f32, example_data_f32);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Destination tensor to write the 8 bit integer quantized values in</span></div>
<div class="line">int8_t example_data_q7[2*3];</div>
<div class="line">uint16_t example_shape_q7[] = {2, 3};</div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> example_q_params_q7;</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> example_tensor_q7 = AITENSOR_2D_Q7(example_shape_q7, &amp;example_q_params_q7, example_data_q7);</div>
<div class="line"> </div>
<div class="line"><span class="keywordtype">float</span> min_value, max_value;</div>
<div class="line"><a class="code" href="aimath__f32__default_8h.html#a29b2f81f252312acbac15df0d415d791">aimath_f32_default_min</a>(&amp;example_tensor_f32, &amp;min_value);</div>
<div class="line"><a class="code" href="aimath__f32__default_8h.html#a2cca0f45a41c48a7730664f43933c066">aimath_f32_default_max</a>(&amp;example_tensor_f32, &amp;max_value);</div>
<div class="line"><a class="code" href="aimath__q7_8h.html#a8fba8e004fef67ff8ac8689002bdfaaa">aimath_q7_calc_q_params_from_f32</a>(min_value, max_value, &amp;example_q_params_q7); <span class="comment">// Calculate the quantization parameters for a tensor with values in the given range</span></div>
<div class="line"><a class="code" href="aimath__q7_8h.html#a1a7229c2ed55114c37a0e30d74346b1a">aimath_q7_quantize_tensor_from_f32</a>(&amp;example_tensor_f32, &amp;example_tensor_q7); <span class="comment">// Quantize the F32 tensor and write the results to the Q7 tensor</span></div>
<div class="line"> </div>
<div class="line"><span class="comment">// Print the quantized tensor to the console</span></div>
<div class="line"><a class="code" href="aimath__basic_8h.html#ab10c8d06990943806f0be8fcc6af03fc">print_aitensor</a>(&amp;example_tensor_q7);</div>
<div class="ttc" id="aaimath__basic_8h_html_ab10c8d06990943806f0be8fcc6af03fc"><div class="ttname"><a href="aimath__basic_8h.html#ab10c8d06990943806f0be8fcc6af03fc">print_aitensor</a></div><div class="ttdeci">void print_aitensor(const aitensor_t *tensor)</div><div class="ttdoc">Printing a tensor to console.</div></div>
<div class="ttc" id="aaimath__f32__default_8h_html_a29b2f81f252312acbac15df0d415d791"><div class="ttname"><a href="aimath__f32__default_8h.html#a29b2f81f252312acbac15df0d415d791">aimath_f32_default_min</a></div><div class="ttdeci">void aimath_f32_default_min(const aitensor_t *x, void *result)</div><div class="ttdoc">Identifies the minimum value in a F32  tensor.</div></div>
<div class="ttc" id="aaimath__f32__default_8h_html_a2cca0f45a41c48a7730664f43933c066"><div class="ttname"><a href="aimath__f32__default_8h.html#a2cca0f45a41c48a7730664f43933c066">aimath_f32_default_max</a></div><div class="ttdeci">void aimath_f32_default_max(const aitensor_t *x, void *result)</div><div class="ttdoc">Identifies the maximum value in a F32  tensor.</div></div>
<div class="ttc" id="aaimath__q7_8h_html_a1a7229c2ed55114c37a0e30d74346b1a"><div class="ttname"><a href="aimath__q7_8h.html#a1a7229c2ed55114c37a0e30d74346b1a">aimath_q7_quantize_tensor_from_f32</a></div><div class="ttdeci">void aimath_q7_quantize_tensor_from_f32(const aitensor_t *tensor_f32, aitensor_t *tensor_q7)</div><div class="ttdoc">Converts a float f32 tensor into a quantized q7 tensor.</div></div>
<div class="ttc" id="aaimath__q7_8h_html_a8fba8e004fef67ff8ac8689002bdfaaa"><div class="ttname"><a href="aimath__q7_8h.html#a8fba8e004fef67ff8ac8689002bdfaaa">aimath_q7_calc_q_params_from_f32</a></div><div class="ttdeci">void aimath_q7_calc_q_params_from_f32(float min_value, float max_value, aimath_q7_params_t *q_params)</div><div class="ttdoc">Calculates the aimath_q7_params parameters.</div></div>
</div><!-- fragment --><p>Symmetric quantization (zero point = 0): </p><div class="fragment"><div class="line"><span class="comment">// Source tensor with 32 bit float values</span></div>
<div class="line"><span class="keywordtype">float</span> example_data_f32[] = {0.0f, 1.0f, 2.0f,</div>
<div class="line">                            3.0f, 4.0f, 5.0f};</div>
<div class="line">uint16_t example_shape_f32[] = {2, 3};</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> example_tensor_f32 = AITENSOR_2D_F32(example_shape_f32, example_data_f32);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Destination tensor to write the 8 bit integer quantized values in</span></div>
<div class="line">int8_t example_data_q7[2*3];</div>
<div class="line">uint16_t example_shape_q7[] = {2, 3};</div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> example_q_params_q7;</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> example_tensor_q7 = AITENSOR_2D_Q7(example_shape_q7, &amp;example_q_params_q7, example_data_q7);</div>
<div class="line"> </div>
<div class="line"><span class="keywordtype">float</span> min_value, max_value;</div>
<div class="line"><a class="code" href="aimath__f32__default_8h.html#a29b2f81f252312acbac15df0d415d791">aimath_f32_default_min</a>(&amp;example_tensor_f32, &amp;min_value);</div>
<div class="line"><a class="code" href="aimath__f32__default_8h.html#a2cca0f45a41c48a7730664f43933c066">aimath_f32_default_max</a>(&amp;example_tensor_f32, &amp;max_value);</div>
<div class="line"><span class="keywordflow">if</span>(max_value &lt; -min_value){</div>
<div class="line">    max_value = -min_value;</div>
<div class="line">}</div>
<div class="line"><a class="code" href="aimath__q7_8h.html#a8fba8e004fef67ff8ac8689002bdfaaa">aimath_q7_calc_q_params_from_f32</a>(-max_value, max_value, &amp;example_q_params_q7); <span class="comment">// Calculate the quantization parameters for a tensor with values in the given range</span></div>
<div class="line"><a class="code" href="aimath__q7_8h.html#a1a7229c2ed55114c37a0e30d74346b1a">aimath_q7_quantize_tensor_from_f32</a>(&amp;example_tensor_f32, &amp;example_tensor_q7); <span class="comment">// Quantize the F32 tensor and write the results to the Q7 tensor</span></div>
<div class="line"> </div>
<div class="line"><span class="comment">// Print the quantized tensor to the console</span></div>
<div class="line"><a class="code" href="aimath__basic_8h.html#ab10c8d06990943806f0be8fcc6af03fc">print_aitensor</a>(&amp;example_tensor_q7);</div>
</div><!-- fragment --><h1><a class="anchor" id="autotoc_md36"></a>
Manual quantization of the neural network</h1>
<h2><a class="anchor" id="autotoc_md37"></a>
Quantization of the intermediate results</h2>
<p>To perform integer-only calculations, the first thing of an ANN we need to quantize are the intermediate results of the layers. Every layer needs to know in which range the values of its result are, in order to do the right calculations without any overflow. Therefor we have to perform some inferences on a representative part of the dataset (or the whole dataset) and remember the min and max values of each layers result tensor. Afterwards we can calculate the quantization parameters (shift and zero point) and configure them to the result tensors.</p>
<p><b>Example:</b></p>
<p>Our input dataset is given by the tensor </p><p class="formulaDsp">
\[ \left( \begin{array}{rrr} 0 &amp; 0 &amp; 0 \\ 1 &amp; 1 &amp; 1 \\ 1 &amp; 0 &amp; 0 \end{array}\right) \]
</p>
<p>The inference / forward pass for the layers gives us the following values:</p>
<p>Dense 1: </p><p class="formulaDsp">
\[ \left( \begin{array}{rrr} 0.72655 &amp; 2.67281 &amp; -0.21291 \\ -1.38335 &amp; -4.09500 &amp; 5.13747 \\ 4.37195 &amp; -0.93700 &amp; 1.36340 \end{array}\right) \]
</p>
<p>Leaky ReLU: </p><p class="formulaDsp">
\[ \left( \begin{array}{rrr} 0.72655 &amp; 2.67281 &amp; -0.00213 \\ -0.01383 &amp; -0.04095 &amp; 5.13747 \\ 4.37195 &amp; -0.00937 &amp; 1.36340 \end{array}\right) \]
</p>
<p>Dense 2: </p><p class="formulaDsp">
\[ \left( \begin{array}{rrr} 8.00280 &amp; -10.82488 \\ -12.87884 &amp; 11.78870 \\ -8.08759 &amp; -8.56308 \end{array}\right) \]
</p>
<p>Sigmoid: </p><p class="formulaDsp">
\[ \left( \begin{array}{rrr} 0.99967 &amp; 0.00002 \\ 0.00000 &amp; 0.99999 \\ 0.00031 &amp; 0.00019 \end{array}\right) \]
</p>
<p>Now we can calculate the quantization parameters (shift and zero point) with the value range (minimum and maximum values) of the result tensors of the input and dense layers. (The activation layers can calculate the quantization parameters on their own, because they just perform fixed transformations. Check the documentation to see if a layer needs manual calculation of the parameters or not.) For this we can use the <a class="el" href="aimath__q7_8h.html#a8fba8e004fef67ff8ac8689002bdfaaa" title="Calculates the aimath_q7_params parameters.">aimath_q7_calc_q_params_from_f32()</a> function. It is recommendet to add a small safety margin to the min and max values to deal with unseen data that results in slightly bigger values.</p>
<p>The min and max values and the resulting quantization parameters of the intermediate results of the layers are:</p>
<table class="markdownTable">
<tr class="markdownTableHead">
<th class="markdownTableHeadNone">layer   </th><th class="markdownTableHeadNone">min   </th><th class="markdownTableHeadNone">max   </th><th class="markdownTableHeadNone">margin   </th><th class="markdownTableHeadNone">shift   </th><th class="markdownTableHeadNone">zero point    </th></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">Input   </td><td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">1   </td><td class="markdownTableBodyNone">10 %   </td><td class="markdownTableBodyNone">7   </td><td class="markdownTableBodyNone">-70    </td></tr>
<tr class="markdownTableRowEven">
<td class="markdownTableBodyNone">Dense 1   </td><td class="markdownTableBodyNone">-4.095   </td><td class="markdownTableBodyNone">5.13747   </td><td class="markdownTableBodyNone">10 %   </td><td class="markdownTableBodyNone">4   </td><td class="markdownTableBodyNone">-9    </td></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">Dense 2   </td><td class="markdownTableBodyNone">-12.878839   </td><td class="markdownTableBodyNone">11.788695   </td><td class="markdownTableBodyNone">10 %   </td><td class="markdownTableBodyNone">3   </td><td class="markdownTableBodyNone">5   </td></tr>
</table>
<p>Example for quantization parameter calculation with margin: </p><div class="fragment"><div class="line"><span class="keywordtype">float</span> min_value = 0.0f;</div>
<div class="line"><span class="keywordtype">float</span> max_value = 1.0f;</div>
<div class="line"><span class="keywordtype">float</span> margin = 0.1f;</div>
<div class="line"> </div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> example_q_params;</div>
<div class="line"> </div>
<div class="line"><a class="code" href="aimath__q7_8h.html#a8fba8e004fef67ff8ac8689002bdfaaa">aimath_q7_calc_q_params_from_f32</a>(min_value * (1.0f + margin), max_value * (1.0f + margin), &amp;example_q_params);</div>
</div><!-- fragment --><h2><a class="anchor" id="autotoc_md38"></a>
Quantization of the weights and biases</h2>
<p>In our simple example, we have two dense layers with weights and biases. For the integer model, we need to quantize the floating point values (given in <a class="el" href="_tutorial_inference_q7.html#GetWeightsAndBiases">the section above</a>) to the integer type. To simplify the calculations of the inference, we use the symmetric quantization (zero point is zero). We perform a 8 bit integer quantization (Q7) on the weights and a 32 bit integer quantization (Q31) on the bias, to get an efficient tradeoff between performance, size and accuracy.</p>
<p>The maximum absolute value of the weight tensors and the resulting quantization parameters are</p>
<table class="markdownTable">
<tr class="markdownTableHead">
<th class="markdownTableHeadNone">tensor   </th><th class="markdownTableHeadNone">max_abs   </th><th class="markdownTableHeadNone">shift   </th><th class="markdownTableHeadNone">zero point    </th></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">Weights 1   </td><td class="markdownTableBodyNone">3.6454   </td><td class="markdownTableBodyNone">5   </td><td class="markdownTableBodyNone">0    </td></tr>
<tr class="markdownTableRowEven">
<td class="markdownTableBodyNone">Weights 2   </td><td class="markdownTableBodyNone">3.23528   </td><td class="markdownTableBodyNone">5   </td><td class="markdownTableBodyNone">0   </td></tr>
</table>
<p>The shift of the (32 bit quantized) bias tensor is the sum of the related weights shift and result shift of the previous layer (This is due to the simple addition of the bias value to the internal accumulator without shift correction) are</p>
<table class="markdownTable">
<tr class="markdownTableHead">
<th class="markdownTableHeadNone">tensor   </th><th class="markdownTableHeadNone">weights shift of same layer   </th><th class="markdownTableHeadNone">result shift of previous layer   </th><th class="markdownTableHeadNone">shift   </th><th class="markdownTableHeadNone">zero point    </th></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">Bias 1   </td><td class="markdownTableBodyNone">5   </td><td class="markdownTableBodyNone">7   </td><td class="markdownTableBodyNone">12   </td><td class="markdownTableBodyNone">0    </td></tr>
<tr class="markdownTableRowEven">
<td class="markdownTableBodyNone">Bias 2   </td><td class="markdownTableBodyNone">5   </td><td class="markdownTableBodyNone">4   </td><td class="markdownTableBodyNone">9   </td><td class="markdownTableBodyNone">0   </td></tr>
</table>
<p>Tipp: Check the documentation of the layers in the Q7 datatype (e.g. of <a class="el" href="ailayer__sigmoid__default_8h.html#ad6c8e7d5957c33998ab12847b651e263" title="Initializes and connect a Sigmoid layer  with the Q7  default implementation.">ailayer_sigmoid_q7_default()</a>) to find the result shift of the activation layers. In addition, the function <a class="el" href="structailayer.html#ab5e8fd65efe85a1ee1ac9147594ddd61" title="If available, calculate and set the tensor_params of the result tensor.">ailayer.calc_result_tensor_params</a> can calculate the value.</p>
<p>The resulting integer values of the quantized weights and bias tensors are</p>
<p class="formulaDsp">
\[ w_1 = \left( \begin{array}{c} 117 &amp; -116 &amp; 50 \\ -96 &amp; -61 &amp; 98 \\ -89 &amp; -40 &amp; 23 \end{array}\right) \]
</p>
 <p class="formulaDsp">
\[ b_1 = \left( \begin{array}{c} 2976 &amp; 10948 &amp; -872 \end{array}\right) \]
</p>
 <p class="formulaDsp">
\[ w_2 = \left( \begin{array}{c} -35 &amp; -78 \\ 104 &amp; -92 \\ -80 &amp; 81 \end{array}\right) \]
</p>
 <p class="formulaDsp">
\[ b_2 = \left( \begin{array}{c} 74 &amp; -688 \end{array}\right) \]
</p>
<h1><a class="anchor" id="autotoc_md39"></a>
Create the neural network in AIfES</h1>
<p>AIfES provides implementations of the layers for different data types that can be optimized for several hardware platforms. An overview of the layers that are available for inference can be seen in the <a class="el" href="index.html#OverviewInference">overview section of the main page</a>. In the overview table you can click on the layer in the first column for a description on how the layer works. To see how to create the layer in code, choose one of the implementations for your used data type and click on the link. <br  />
 In this tutorial we work with the int 8 data type (<a class="el" href="aimath__q7_8h.html">Q7 </a>) and use the default implementations (without any hardware specific optimizations) of the layers.</p>
<p>Used layer types:</p><ul>
<li><a class="el" href="ailayer__input_8h.html">Input layer </a></li>
<li><a class="el" href="ailayer__dense_8h.html">Dense layer </a></li>
<li><a class="el" href="ailayer__leaky__relu_8h.html">Leaky ReLU layer </a></li>
<li><a class="el" href="ailayer__sigmoid_8h.html">Sigmoid layer </a></li>
</ul>
<p>Used implementations:</p><ul>
<li><a class="el" href="ailayer__input__default_8h.html#a20e15725d131ac1488c236994add73dd" title="Initializes and connect an Input layer  with the Q7  default implementation.">ailayer_input_q7_default()</a></li>
<li><a class="el" href="ailayer__dense__default_8h.html#a039ecc8141aac279c2d09d79443a2717" title="Initializes and connect a Dense layer  with the Q7  default implementation.">ailayer_dense_q7_default()</a></li>
<li><a class="el" href="ailayer__leaky__relu__default_8h.html#a85b42fd39716f57363a770edca84381d" title="Initializes and connect a Leaky ReLU layer  with the Q7  default implementation.">ailayer_leaky_relu_q7_default()</a></li>
<li><a class="el" href="ailayer__sigmoid__default_8h.html#ad6c8e7d5957c33998ab12847b651e263" title="Initializes and connect a Sigmoid layer  with the Q7  default implementation.">ailayer_sigmoid_q7_default()</a></li>
</ul>
<h2><a class="anchor" id="ManualLayerDeclaration"></a>
Manual declaration and configuration of the layers</h2>
<p>For every layer we need to create a variable of the specific layer type and configure it for our needs. See the documentation of the data type and hardware specific implementations (for example <a class="el" href="ailayer__dense__default_8h.html#a039ecc8141aac279c2d09d79443a2717" title="Initializes and connect a Dense layer  with the Q7  default implementation.">ailayer_dense_q7_default()</a>) for code examples on how to configure the layers.</p>
<p>Our designed network can be declared with the following code. </p><div class="fragment"><div class="line"><span class="comment">// The main model structure that holds the whole neural network</span></div>
<div class="line"><a class="code" href="structaimodel.html">aimodel_t</a> model;</div>
<div class="line"> </div>
<div class="line"> </div>
<div class="line"><span class="comment">// The layer structures for Q7 data type and their configurations</span></div>
<div class="line"> </div>
<div class="line"><span class="comment">// Input</span></div>
<div class="line">uint16_t input_layer_shape[] = {1, 3};</div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> input_layer_result_q_params = { .<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 7, .zero_point = -70 };</div>
<div class="line"><a class="code" href="structailayer__input.html">ailayer_input_q7_t</a> input_layer = AILAYER_INPUT_Q7_M(2, input_layer_shape, &amp;input_layer_result_q_params);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Dense</span></div>
<div class="line"><span class="comment">// Weights (8 bit quantized)</span></div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> dense_layer_1_weights_q_params = { .<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 5, .zero_point = 0 };</div>
<div class="line">int8_t dense_layer_1_weights[3*3] = {117, -116, 50,</div>
<div class="line">                                     -96,  -61, 98,</div>
<div class="line">                                     -89,  -40, 23};</div>
<div class="line"><span class="comment">// Bias (32 bit quantized)</span></div>
<div class="line"><a class="code" href="structaimath__q31__params.html">aimath_q31_params_t</a> dense_layer_1_bias_q_params = { .<a class="code" href="structaimath__q31__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 12, .zero_point = 0 };</div>
<div class="line">int32_t dense_layer_1_bias[1*3] = {2976, 10948, -872};</div>
<div class="line"><span class="comment">// Result (8 bit quantized)</span></div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> dense_layer_1_result_q_params = { .<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 4, .zero_point = -9 };</div>
<div class="line"><a class="code" href="structailayer__dense.html">ailayer_dense_q7_t</a> dense_layer_1 = AILAYER_DENSE_Q7_M(3,</div>
<div class="line">                                                        dense_layer_1_weights, &amp;dense_layer_1_weights_q_params,</div>
<div class="line">                                                        dense_layer_1_bias, &amp;dense_layer_1_bias_q_params,</div>
<div class="line">                                                        &amp;dense_layer_1_result_q_params);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Leaky ReLU</span></div>
<div class="line"><span class="comment">// Result (8 bit quantized)</span></div>
<div class="line"><a class="code" href="structailayer__leaky__relu__q7.html">ailayer_leaky_relu_q7_t</a> leaky_relu_layer = AILAYER_LEAKY_RELU_Q7_M(AISCALAR_Q7(0.01f,10,0));</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Dense</span></div>
<div class="line"><span class="comment">// Weights (8 bit quantized)</span></div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> dense_layer_2_weights_q_params = { .<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 5, .zero_point = 0 };</div>
<div class="line">int8_t dense_layer_2_weights[3*2] = {-35, -78,</div>
<div class="line">                                     104, -92,</div>
<div class="line">                                     -80,  81};</div>
<div class="line"><span class="comment">// Bias (32 bit quantized)</span></div>
<div class="line"><a class="code" href="structaimath__q31__params.html">aimath_q31_params_t</a> dense_layer_2_bias_q_params = { .<a class="code" href="structaimath__q31__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 9, .zero_point = 0 };</div>
<div class="line">int32_t dense_layer_2_bias[1*2] = {74, -688};</div>
<div class="line"><span class="comment">// Result (8 bit quantized)</span></div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> dense_layer_2_result_q_params = { .<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 3, .zero_point = 5 };</div>
<div class="line"><a class="code" href="structailayer__dense.html">ailayer_dense_q7_t</a> dense_layer_2 = AILAYER_DENSE_Q7_M(2,</div>
<div class="line">                                                        dense_layer_2_weights, &amp;dense_layer_2_weights_q_params,</div>
<div class="line">                                                        dense_layer_2_bias, &amp;dense_layer_2_bias_q_params,</div>
<div class="line">                                                        &amp;dense_layer_2_result_q_params);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Sigmoid</span></div>
<div class="line"><span class="comment">// Result (8 bit quantized)</span></div>
<div class="line"><a class="code" href="structailayer__sigmoid.html">ailayer_sigmoid_q7_t</a> sigmoid_layer = AILAYER_SIGMOID_Q7_M();</div>
<div class="ttc" id="astructailayer__dense_html"><div class="ttname"><a href="structailayer__dense.html">ailayer_dense</a></div><div class="ttdoc">General Dense layer  structure.</div><div class="ttdef"><b>Definition:</b> ailayer_dense.h:71</div></div>
<div class="ttc" id="astructailayer__input_html"><div class="ttname"><a href="structailayer__input.html">ailayer_input</a></div><div class="ttdoc">General Input layer  structure.</div><div class="ttdef"><b>Definition:</b> ailayer_input.h:39</div></div>
<div class="ttc" id="astructailayer__leaky__relu__q7_html"><div class="ttname"><a href="structailayer__leaky__relu__q7.html">ailayer_leaky_relu_q7</a></div><div class="ttdoc">Data-type specific Leaky ReLU layer struct for Q7 .</div><div class="ttdef"><b>Definition:</b> ailayer_leaky_relu_default.h:69</div></div>
<div class="ttc" id="astructailayer__sigmoid_html"><div class="ttname"><a href="structailayer__sigmoid.html">ailayer_sigmoid</a></div><div class="ttdoc">General Sigmoid layer  struct.</div><div class="ttdef"><b>Definition:</b> ailayer_sigmoid.h:47</div></div>
<div class="ttc" id="astructaimath__q31__params_html"><div class="ttname"><a href="structaimath__q31__params.html">aimath_q31_params</a></div><div class="ttdoc">Parameters used for the quantized Q31  values, used as property of a tensor.</div><div class="ttdef"><b>Definition:</b> aimath_q31.h:149</div></div>
<div class="ttc" id="astructaimath__q31__params_html_a8f20d77488fe485aaf9ea8e9c5d7d35e"><div class="ttname"><a href="structaimath__q31__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">aimath_q31_params::shift</a></div><div class="ttdeci">uint16_t shift</div><div class="ttdoc">The scaling factor  of the quantization (The total scale is calculated with )</div><div class="ttdef"><b>Definition:</b> aimath_q31.h:150</div></div>
<div class="ttc" id="astructaimodel_html"><div class="ttname"><a href="structaimodel.html">aimodel</a></div><div class="ttdoc">AIfES artificial neural network model.</div><div class="ttdef"><b>Definition:</b> aifes_core.h:181</div></div>
</div><!-- fragment --><p> We use the initializer macros with the "_M" (for "Manually") at the end, because we need to set our parameters (like the weights) to the layers.</p>
<h2><a class="anchor" id="AutomaticLayerDeclaration"></a>
Automatic declaration and configuration of the layers</h2>
<p>If you already have the parameters of your model as a flat byte array (for example by <a class="el" href="_tutorial_inference_q7.html#AutomaticQuantizationPython">quantization in python using the aifes-pytools</a> or by loading the parameters from a stored <a class="el" href="_tutorial_inference_q7.html#AutomaticQuantizationAIfES">buffer created with AIfES</a>), you can use the automatic configuration of the layer. This way makes it easier to configure the neural network and to update new weight sets. On the other hand you lose some flexibility on where you store your data and it is harder to distinguish between the different values. Also the buffer format might change slightly with further updates of AIfES, so make shure that everything works out when updating.</p>
<div class="fragment"><div class="line"><span class="comment">// The main model structure that holds the whole neural network</span></div>
<div class="line"><a class="code" href="structaimodel.html">aimodel_t</a> model;</div>
<div class="line"> </div>
<div class="line"><span class="comment">// The layer structures for Q7 data type and their configurations</span></div>
<div class="line"> </div>
<div class="line">uint16_t input_layer_shape[] = {1, 3};</div>
<div class="line"><a class="code" href="structailayer__input.html">ailayer_input_q7_t</a> input_layer           = AILAYER_INPUT_Q7_A(2, input_layer_shape);</div>
<div class="line"><a class="code" href="structailayer__dense.html">ailayer_dense_q7_t</a> dense_layer_1         = AILAYER_DENSE_Q7_A(3);</div>
<div class="line"><a class="code" href="structailayer__leaky__relu__q7.html">ailayer_leaky_relu_q7_t</a> leaky_relu_layer = AILAYER_LEAKY_RELU_Q7_A(AISCALAR_Q7(0.01f,10,0));</div>
<div class="line"><a class="code" href="structailayer__dense.html">ailayer_dense_q7_t</a> dense_layer_2         = AILAYER_DENSE_Q7_A(2);</div>
<div class="line"><a class="code" href="structailayer__sigmoid.html">ailayer_sigmoid_q7_t</a> sigmoid_layer       = AILAYER_SIGMOID_Q7_A();</div>
<div class="line"> </div>
<div class="line"><span class="comment">// ... do the layer connetion and model compilation of the following chapter here ...</span></div>
<div class="line"> </div>
<div class="line"> </div>
<div class="line"><span class="comment">// The parameter memory may vary on different architectures because the parameters have to be memory aligned</span></div>
<div class="line"><span class="preprocessor">#define MEMORY_ALIGNMENT    4   </span><span class="comment">// Must have the same size as AIFES_MEMORY_ALIGNMENT</span></div>
<div class="line"><span class="preprocessor">#if MEMORY_ALIGNMENT == 4</span></div>
<div class="line"> </div>
<div class="line"><span class="comment">// This is an example parameter buffer for a memory alignment of 4 byte where integer are stored with little endian byteorder</span></div>
<div class="line"><span class="keyword">const</span> uint32_t parameter_memory_size = 76;</div>
<div class="line"><span class="keyword">const</span> uint32_t model_parameters[ 19 ] = {</div>
<div class="line">    0x00BA0007, 0x00F70004, 0x00050003, 0x00000005, 0xA0328C75, 0xD8A762C3, 0x00000017, 0x0000000C,</div>
<div class="line">    0x00000000, 0x00000BA0, 0x00002AC4, 0xFFFFFC98, 0x00000005, 0xA468B2DD, 0x000051B0, 0x00000009,</div>
<div class="line">    0x00000000, 0x0000004A, 0xFFFFFD50</div>
<div class="line">};</div>
<div class="line"> </div>
<div class="line"><span class="preprocessor">#elif MEMORY_ALIGNMENT == 2</span></div>
<div class="line"> </div>
<div class="line"><span class="comment">// This is an example parameter buffer for a memory alignment of 2 byte where integer are stored with little endian byteorder</span></div>
<div class="line"><span class="keyword">const</span> uint32_t parameter_memory_size = 68;</div>
<div class="line"><span class="keyword">const</span> uint16_t model_parameters[ 34 ] = {</div>
<div class="line">    0x0007, 0x00BA, 0x0004, 0x00F7, 0x0003, 0x0005, 0x0005, 0x0000, 0x8C75, 0xA032, 0x62C3, 0xD8A7, 0x0017, 0x000C, 0x0000, 0x0000,</div>
<div class="line">    0x0BA0, 0x0000, 0x2AC4, 0x0000, 0xFC98, 0xFFFF, 0x0005, 0x0000, 0xB2DD, 0xA468, 0x51B0, 0x0009, 0x0000, 0x0000, 0x004A, 0x0000,</div>
<div class="line">    0xFD50, 0xFFFF</div>
<div class="line">};</div>
<div class="line"> </div>
<div class="line"><span class="preprocessor">#endif</span></div>
<div class="line"> </div>
<div class="line"><span class="comment">// Set the model parameters to the layers of the model</span></div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a26db68cb4231b534b03c649d6eeab3f8">aialgo_distribute_parameter_memory</a>(&amp;model, model_parameters, parameter_memory_size);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a26db68cb4231b534b03c649d6eeab3f8"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a26db68cb4231b534b03c649d6eeab3f8">aialgo_distribute_parameter_memory</a></div><div class="ttdeci">void aialgo_distribute_parameter_memory(aimodel_t *model, void *memory_ptr, uint32_t memory_size)</div><div class="ttdoc">Assign the memory for the trainable parameters (like weights, bias, ...) of the model.</div></div>
</div><!-- fragment --><p> We use the initializer macros with the "_A" (for "Automatically") at the end, because our parameters (like the weights) will be set by an aifes function to the layers.</p>
<h2><a class="anchor" id="autotoc_md40"></a>
Connection and initialization of the layers</h2>
<p>Afterwards the layers are connected and initialized with the data type and hardware specific implementations </p><div class="fragment"><div class="line"><span class="comment">// Layer pointer to perform the connection</span></div>
<div class="line"><a class="code" href="structailayer.html">ailayer_t</a> *x;</div>
<div class="line"> </div>
<div class="line">model.<a class="code" href="structaimodel.html#a708a94e69112ad215b2b52da2238a711">input_layer</a> = <a class="code" href="ailayer__input__default_8h.html#a20e15725d131ac1488c236994add73dd">ailayer_input_q7_default</a>(&amp;input_layer);</div>
<div class="line">x = <a class="code" href="ailayer__dense__default_8h.html#a039ecc8141aac279c2d09d79443a2717">ailayer_dense_q7_default</a>(&amp;dense_layer_1, model.<a class="code" href="structaimodel.html#a708a94e69112ad215b2b52da2238a711">input_layer</a>);</div>
<div class="line">x = <a class="code" href="ailayer__leaky__relu__default_8h.html#a85b42fd39716f57363a770edca84381d">ailayer_leaky_relu_q7_default</a>(&amp;leaky_relu_layer, x);</div>
<div class="line">x = <a class="code" href="ailayer__dense__default_8h.html#a039ecc8141aac279c2d09d79443a2717">ailayer_dense_q7_default</a>(&amp;dense_layer_2, x);</div>
<div class="line">x = <a class="code" href="ailayer__sigmoid__default_8h.html#ad6c8e7d5957c33998ab12847b651e263">ailayer_sigmoid_q7_default</a>(&amp;sigmoid_layer, x);</div>
<div class="line">model.<a class="code" href="structaimodel.html#a7c7ad89e7d15631b3f5893b8f19030ef">output_layer</a> = x;</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Finish the model creation by checking the connections and setting some parameters for further processing</span></div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a3fb665166082f1e7a89e23218a105ce8">aialgo_compile_model</a>(&amp;model);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a3fb665166082f1e7a89e23218a105ce8"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a3fb665166082f1e7a89e23218a105ce8">aialgo_compile_model</a></div><div class="ttdeci">uint8_t aialgo_compile_model(aimodel_t *model)</div><div class="ttdoc">Initialize the model structure.</div></div>
<div class="ttc" id="aailayer__dense__default_8h_html_a039ecc8141aac279c2d09d79443a2717"><div class="ttname"><a href="ailayer__dense__default_8h.html#a039ecc8141aac279c2d09d79443a2717">ailayer_dense_q7_default</a></div><div class="ttdeci">ailayer_t * ailayer_dense_q7_default(ailayer_dense_q7_t *layer, ailayer_t *input_layer)</div><div class="ttdoc">Initializes and connect a Dense layer  with the Q7  default implementation.</div></div>
<div class="ttc" id="aailayer__input__default_8h_html_a20e15725d131ac1488c236994add73dd"><div class="ttname"><a href="ailayer__input__default_8h.html#a20e15725d131ac1488c236994add73dd">ailayer_input_q7_default</a></div><div class="ttdeci">ailayer_t * ailayer_input_q7_default(ailayer_input_q7_t *layer)</div><div class="ttdoc">Initializes and connect an Input layer  with the Q7  default implementation.</div></div>
<div class="ttc" id="aailayer__leaky__relu__default_8h_html_a85b42fd39716f57363a770edca84381d"><div class="ttname"><a href="ailayer__leaky__relu__default_8h.html#a85b42fd39716f57363a770edca84381d">ailayer_leaky_relu_q7_default</a></div><div class="ttdeci">ailayer_t * ailayer_leaky_relu_q7_default(ailayer_leaky_relu_q7_t *layer, ailayer_t *input_layer)</div><div class="ttdoc">Initializes and connect a Leaky ReLU layer  with the Q7  default implementation.</div></div>
<div class="ttc" id="aailayer__sigmoid__default_8h_html_ad6c8e7d5957c33998ab12847b651e263"><div class="ttname"><a href="ailayer__sigmoid__default_8h.html#ad6c8e7d5957c33998ab12847b651e263">ailayer_sigmoid_q7_default</a></div><div class="ttdeci">ailayer_t * ailayer_sigmoid_q7_default(ailayer_sigmoid_q7_t *layer, ailayer_t *input_layer)</div><div class="ttdoc">Initializes and connect a Sigmoid layer  with the Q7  default implementation.</div></div>
<div class="ttc" id="astructailayer_html"><div class="ttname"><a href="structailayer.html">ailayer</a></div><div class="ttdoc">AIfES layer interface.</div><div class="ttdef"><b>Definition:</b> aifes_core.h:252</div></div>
<div class="ttc" id="astructaimodel_html_a708a94e69112ad215b2b52da2238a711"><div class="ttname"><a href="structaimodel.html#a708a94e69112ad215b2b52da2238a711">aimodel::input_layer</a></div><div class="ttdeci">ailayer_t * input_layer</div><div class="ttdoc">Input layer of the model that gets the input data.</div><div class="ttdef"><b>Definition:</b> aifes_core.h:182</div></div>
<div class="ttc" id="astructaimodel_html_a7c7ad89e7d15631b3f5893b8f19030ef"><div class="ttname"><a href="structaimodel.html#a7c7ad89e7d15631b3f5893b8f19030ef">aimodel::output_layer</a></div><div class="ttdeci">ailayer_t * output_layer</div><div class="ttdoc">Output layer of the model.</div><div class="ttdef"><b>Definition:</b> aifes_core.h:183</div></div>
</div><!-- fragment --><h2><a class="anchor" id="autotoc_md41"></a>
Print the layer structure to the console</h2>
<p>To see the structure of your created model, you can print a model summary to the console </p><div class="fragment"><div class="line">aiprint(<span class="stringliteral">&quot;\n-------------- Model structure ---------------\n&quot;</span>);</div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a3bb5fdb556c51ad8b3043d220f0a1276">aialgo_print_model_structure</a>(&amp;model);</div>
<div class="line">aiprint(<span class="stringliteral">&quot;----------------------------------------------\n\n&quot;</span>);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a3bb5fdb556c51ad8b3043d220f0a1276"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a3bb5fdb556c51ad8b3043d220f0a1276">aialgo_print_model_structure</a></div><div class="ttdeci">void aialgo_print_model_structure(aimodel_t *model)</div><div class="ttdoc">Print the layer structure of the model with the configured parameters.</div></div>
</div><!-- fragment --><h1><a class="anchor" id="autotoc_md42"></a>
Perform the inference</h1>
<h2><a class="anchor" id="autotoc_md43"></a>
Allocate and initialize the working memory</h2>
<p>Because AIfES doesn't allocate any memory on its own, you have to set the memory buffers for the inference manually. This memory is required for example for the intermediate results of the layers. Therefore you can choose fully flexible, where the buffer should be located in memory. To calculate the required amount of memory for the inference, the <a class="el" href="aialgo__sequential__inference_8h.html#a877ce6eee19a9f9bcbdb115d83537e68" title="Calculate the memory requirements for intermediate results of an inference.">aialgo_sizeof_inference_memory()</a> function can be used. <br  />
 With <a class="el" href="aialgo__sequential__inference_8h.html#a7cbfac6a46c02107d19af7c8f6e5469a" title="Assign the memory for intermediate results of an inference to the model.">aialgo_schedule_inference_memory()</a> a memory block of the required size can be distributed and scheduled (memory regions might be shared over time) to the model.</p>
<p>A dynamic allocation of the memory using malloc could look like the following: </p><div class="fragment"><div class="line">uint32_t inference_memory_size = <a class="code" href="aialgo__sequential__inference_8h.html#a877ce6eee19a9f9bcbdb115d83537e68">aialgo_sizeof_inference_memory</a>(&amp;model);</div>
<div class="line"><span class="keywordtype">void</span> *inference_memory = malloc(inference_memory_size);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Schedule the memory to the model</span></div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a7cbfac6a46c02107d19af7c8f6e5469a">aialgo_schedule_inference_memory</a>(&amp;model, inference_memory, inference_memory_size);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a7cbfac6a46c02107d19af7c8f6e5469a"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a7cbfac6a46c02107d19af7c8f6e5469a">aialgo_schedule_inference_memory</a></div><div class="ttdeci">uint8_t aialgo_schedule_inference_memory(aimodel_t *model, void *memory_ptr, uint32_t memory_size)</div><div class="ttdoc">Assign the memory for intermediate results of an inference to the model.</div></div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a877ce6eee19a9f9bcbdb115d83537e68"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a877ce6eee19a9f9bcbdb115d83537e68">aialgo_sizeof_inference_memory</a></div><div class="ttdeci">uint32_t aialgo_sizeof_inference_memory(aimodel_t *model)</div><div class="ttdoc">Calculate the memory requirements for intermediate results of an inference.</div></div>
</div><!-- fragment --><p>You could also pre-define a memory buffer if you know the size in advance, for example </p><div class="fragment"><div class="line"><span class="keyword">const</span> uint32_t inference_memory_size = 16;</div>
<div class="line"><span class="keywordtype">char</span> inference_memory[inference_memory_size];</div>
<div class="line"> </div>
<div class="line">...</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Schedule the memory to the model</span></div>
<div class="line">aialgo_schedule_inference_memory(&amp;model, inference_memory, inference_memory_size);</div>
</div><!-- fragment --><h2><a class="anchor" id="RunInferenceQ7"></a>
Run the inference</h2>
<p>To perform the inference, the input data must be packed in a tensor to be processed by AIfES. A tensor in AIfES is just a N-dimensional array that is used to hold the data values in a structured way. To do this in the example, create a 2D tensor of the used data type. The shape describes the size of the dimensions of the tensor. The first dimension (the rows) is the batch dimension, i.e. the dimension of the different input samples. If you process just one sample at a time, this dimension is 1. The second dimension equals the inputs of the neural network.</p>
<div class="fragment"><div class="line">uint16_t in_shape[2] = {1, 3};</div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> in_q_params = { .<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 7, .zero_point = -70 }; <span class="comment">// Same as configured in the input layer</span></div>
<div class="line">int8_t in_data[1*3] = {58, -70, -70};</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> in = AITENSOR_2D_Q7(in_shape, &amp;in_q_params, in_data);</div>
</div><!-- fragment --><p>You can also create a float 32 tensor and let it quantize by AIfES:</p>
<div class="fragment"><div class="line"><span class="comment">// F32 tensor that has to be quantized</span></div>
<div class="line">uint16_t in_f32_shape[2] = {1, 3};</div>
<div class="line"><span class="keywordtype">float</span> in_f32_data[1*3] = {1.0f, 0.0f, 0.0f};</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> in_f32 = AITENSOR_2D_F32(in_f32_shape, in_f32_data);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Automatically quantized Q7 tensor to feed into the neural network</span></div>
<div class="line">uint16_t in_shape[] = {1, 3};</div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> in_q_params = { .<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 7, .zero_point = -70 };</div>
<div class="line">int8_t in_data[1*3];</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> in = AITENSOR_2D_Q7(in_shape, &amp;in_q_params, in_data);</div>
<div class="line"> </div>
<div class="line"><a class="code" href="aimath__q7_8h.html#a1a7229c2ed55114c37a0e30d74346b1a">aimath_q7_quantize_tensor_from_f32</a>(&amp;in_f32, &amp;in); <span class="comment">// Quantize the F32 tensor and write the results to the Q7 tensor</span></div>
</div><!-- fragment --><p>Now everything is ready to perform the actual inference. For this you can use the function <a class="el" href="aialgo__sequential__inference_8h.html#a295cb2bb6c3fefc478042f66e067e090" title="Perform an inference on the model / Run the model.">aialgo_inference_model()</a>.</p>
<div class="fragment"><div class="line"><span class="comment">// Create an empty tensor for the inference results</span></div>
<div class="line">uint16_t out_shape[2] = {1, 2};</div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> out_q_params;</div>
<div class="line"><span class="keywordtype">float</span> out_data[1*2];</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> out = AITENSOR_2D_Q7(out_shape, &amp;out_q_params, out_data);</div>
<div class="line"> </div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a295cb2bb6c3fefc478042f66e067e090">aialgo_inference_model</a>(&amp;model, &amp;in, &amp;out);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a295cb2bb6c3fefc478042f66e067e090"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a295cb2bb6c3fefc478042f66e067e090">aialgo_inference_model</a></div><div class="ttdeci">uint8_t aialgo_inference_model(aimodel_t *model, aitensor_t *input_data, aitensor_t *output_data)</div><div class="ttdoc">Perform an inference on the model / Run the model.</div></div>
</div><!-- fragment --><p>Alternative you can also do the inference without creating an empty tensor for the result with the function <a class="el" href="aialgo__sequential__inference_8h.html#a4655caab3051cc837312c286fbe4789a" title="Perform a forward pass on the model.">aialgo_forward_model()</a>. The results of this function are stored in the inference memory. If you want to perform another inference or delete the inference memory, you have to save the results first to another tensor / array. Otherwise you will loose the data. </p><div class="fragment"><div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> *y = <a class="code" href="aialgo__sequential__inference_8h.html#a4655caab3051cc837312c286fbe4789a">aialgo_forward_model</a>(&amp;model, &amp;in);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a4655caab3051cc837312c286fbe4789a"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a4655caab3051cc837312c286fbe4789a">aialgo_forward_model</a></div><div class="ttdeci">aitensor_t * aialgo_forward_model(aimodel_t *model, aitensor_t *input_data)</div><div class="ttdoc">Perform a forward pass on the model.</div></div>
</div><!-- fragment --><p>Afterwards you can print the results to the console for debugging purposes: </p><div class="fragment"><div class="line">aiprint(<span class="stringliteral">&quot;input:\n&quot;</span>);</div>
<div class="line"><a class="code" href="aimath__basic_8h.html#ab10c8d06990943806f0be8fcc6af03fc">print_aitensor</a>(&amp;in);</div>
<div class="line">aiprint(<span class="stringliteral">&quot;NN output:\n&quot;</span>);</div>
<div class="line"><a class="code" href="aimath__basic_8h.html#ab10c8d06990943806f0be8fcc6af03fc">print_aitensor</a>(&amp;out);</div>
</div><!-- fragment --><p>If you want to get F32 values out of the Q7 tensor, you can use the macro <code>Q7_TO_FLOAT(integer_value, shift, zero_point)</code> as explained earlier: </p><div class="fragment"><div class="line">int8_t value_q7 = -127;</div>
<div class="line"><a class="code" href="structaimath__q7__params.html">aimath_q7_params_t</a> value_q_params = {.<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a> = 8, .zero_point = -128};</div>
<div class="line"> </div>
<div class="line"><span class="keywordtype">float</span> value_f32 = Q7_TO_FLOAT(value_q7, value_q_params.<a class="code" href="structaimath__q7__params.html#a8f20d77488fe485aaf9ea8e9c5d7d35e">shift</a>, value_q_params.<a class="code" href="structaimath__q7__params.html#a6a54cbe2eb02707d6e9ae6f12d539e1b">zero_point</a>);</div>
<div class="ttc" id="astructaimath__q7__params_html_a6a54cbe2eb02707d6e9ae6f12d539e1b"><div class="ttname"><a href="structaimath__q7__params.html#a6a54cbe2eb02707d6e9ae6f12d539e1b">aimath_q7_params::zero_point</a></div><div class="ttdeci">int8_t zero_point</div><div class="ttdoc">The zero point  of the quantization.</div><div class="ttdef"><b>Definition:</b> aimath_q7.h:150</div></div>
</div><!-- fragment --><h1><a class="anchor" id="AutomaticQuantizationAIfES"></a>
Automatic quantization in AIfES</h1>
<p>If you have a trained F32 model in AIfES and you want to convert it to a Q7 quantized model (for example to save memory needed for the weights or to speed up the inference), you can use the provided helper function <a class="el" href="aialgo__sequential__inference_8h.html#a07f280b3565ec1b02f00d907a2834940" title="Quantize model parameters (weights and bias)">aialgo_quantize_model_f32_to_q7()</a>. To use this function, you need to have the F32 model and a Q7 model skeleton with the same structure as the F32 model. Also a dataset is needed that is representative for the input data of the model (for example a fraction or all of the training dataset) to calculate the quantization range.</p>
<p><b>Example</b>:</p>
<p>If you have the example model, described in the <a class="el" href="_tutorial_training_f32.html">F32 training tutorial</a> or the <a class="el" href="_tutorial_inference_f32.html">F32 inference tutorial</a> , you have to build up a Q7 model skeleton that looks like this:</p>
<div class="fragment"><div class="line"><span class="comment">// The main model structure that holds the whole neural network</span></div>
<div class="line"><a class="code" href="structaimodel.html">aimodel_t</a> model_q7;</div>
<div class="line"> </div>
<div class="line"><span class="comment">// The layer structures for Q7 data type and their configurations (Same as in the F32 model, indicated with the &quot;_f32&quot; ending)</span></div>
<div class="line"><a class="code" href="structailayer__input.html">ailayer_input_q7_t</a>      input_layer_q7           = AILAYER_INPUT_Q7_A(2, input_layer_f32.input_shape);</div>
<div class="line"><a class="code" href="structailayer__dense.html">ailayer_dense_q7_t</a>      dense_layer_1_q7         = AILAYER_DENSE_Q7_A(dense_layer_1_f32.neurons);</div>
<div class="line"><a class="code" href="structailayer__leaky__relu__q7.html">ailayer_leaky_relu_q7_t</a> leaky_relu_layer_q7      = AILAYER_LEAKY_RELU_Q7_A(AISCALAR_Q7(leaky_relu_layer_f32.alpha,10,0));</div>
<div class="line"><a class="code" href="structailayer__dense.html">ailayer_dense_q7_t</a>      dense_layer_2_q7         = AILAYER_DENSE_Q7_A(dense_layer_2_f32.neurons);</div>
<div class="line"><a class="code" href="structailayer__sigmoid.html">ailayer_sigmoid_q7_t</a>    sigmoid_layer_q7         = AILAYER_SIGMOID_Q7_A();</div>
<div class="line"> </div>
<div class="line"><a class="code" href="structailayer.html">ailayer_t</a> *x;</div>
<div class="line"> </div>
<div class="line">model_q7.<a class="code" href="structaimodel.html#a708a94e69112ad215b2b52da2238a711">input_layer</a> = <a class="code" href="ailayer__input__default_8h.html#a20e15725d131ac1488c236994add73dd">ailayer_input_q7_default</a>(&amp;input_layer_q7);</div>
<div class="line">x = <a class="code" href="ailayer__dense__default_8h.html#a039ecc8141aac279c2d09d79443a2717">ailayer_dense_q7_default</a>(&amp;dense_layer_1_q7, model_q7.<a class="code" href="structaimodel.html#a708a94e69112ad215b2b52da2238a711">input_layer</a>);</div>
<div class="line">x = <a class="code" href="ailayer__leaky__relu__default_8h.html#a85b42fd39716f57363a770edca84381d">ailayer_leaky_relu_q7_default</a>(&amp;leaky_relu_layer_q7, x);</div>
<div class="line">x = <a class="code" href="ailayer__dense__default_8h.html#a039ecc8141aac279c2d09d79443a2717">ailayer_dense_q7_default</a>(&amp;dense_layer_2_q7, x);</div>
<div class="line">x = <a class="code" href="ailayer__sigmoid__default_8h.html#ad6c8e7d5957c33998ab12847b651e263">ailayer_sigmoid_q7_default</a>(&amp;sigmoid_layer_q7, x);</div>
<div class="line">model_q7.<a class="code" href="structaimodel.html#a7c7ad89e7d15631b3f5893b8f19030ef">output_layer</a> = x;</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Finish the model creation by checking the connections and setting some parameters for further processing</span></div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a3fb665166082f1e7a89e23218a105ce8">aialgo_compile_model</a>(&amp;model_q7);</div>
</div><!-- fragment --><p>In Addition you need to set the parameter memory and the inference memory to the model. In the parameter memory, the quantized parameters (weights, biases, quantization parameters) will be stored and the inference memory is needed to perform the quantization.</p>
<div class="fragment"><div class="line">uint32_t parameter_memory_size_q7 = <a class="code" href="aialgo__sequential__inference_8h.html#acdf3763b8fe9047446ddbcfdbdae5570">aialgo_sizeof_parameter_memory</a>(&amp;model_q7);</div>
<div class="line"><span class="keywordtype">void</span> *parameter_memory_q7 = malloc(parameter_memory_size_q7);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Distribute the memory to the trainable parameters of the model</span></div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a26db68cb4231b534b03c649d6eeab3f8">aialgo_distribute_parameter_memory</a>(&amp;model_q7, parameter_memory_q7, parameter_memory_size_q7);</div>
<div class="line"> </div>
<div class="line">uint32_t inference_memory_size_q7 = <a class="code" href="aialgo__sequential__inference_8h.html#a877ce6eee19a9f9bcbdb115d83537e68">aialgo_sizeof_inference_memory</a>(&amp;model_q7);</div>
<div class="line"><span class="keywordtype">void</span> *inference_memory_q7 = malloc(inference_memory_size_q7);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Schedule the memory to the model</span></div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a7cbfac6a46c02107d19af7c8f6e5469a">aialgo_schedule_inference_memory</a>(&amp;model_q7, inference_memory_q7, inference_memory_size_q7);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_acdf3763b8fe9047446ddbcfdbdae5570"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#acdf3763b8fe9047446ddbcfdbdae5570">aialgo_sizeof_parameter_memory</a></div><div class="ttdeci">uint32_t aialgo_sizeof_parameter_memory(aimodel_t *model)</div><div class="ttdoc">Calculate the memory requirements for the trainable parameters (like weights, bias,...</div></div>
</div><!-- fragment --><p>Then you need the mentioned representative dataset of the input data. In this case we simply take the training dataset because it is very small and has no representative subset.</p>
<div class="fragment"><div class="line">uint16_t x_repr_shape[2] = {3, 3};</div>
<div class="line"><span class="keywordtype">float</span> x_repr_data[3*3] = {0.0f, 0.0f, 0.0f,</div>
<div class="line">                          1.0f, 1.0f, 1.0f,</div>
<div class="line">                          1.0f, 0.0f, 0.0f};</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> x_repr = AITENSOR_2D_F32(x_repr_shape, x_repr_data);</div>
</div><!-- fragment --><p>Now you can perform the quantization by calling <a class="el" href="aialgo__sequential__inference_8h.html#a07f280b3565ec1b02f00d907a2834940" title="Quantize model parameters (weights and bias)">aialgo_quantize_model_f32_to_q7()</a>.</p>
<div class="fragment"><div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a07f280b3565ec1b02f00d907a2834940">aialgo_quantize_model_f32_to_q7</a>(&amp;model_f32, &amp;model_q7, &amp;x_repr);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a07f280b3565ec1b02f00d907a2834940"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a07f280b3565ec1b02f00d907a2834940">aialgo_quantize_model_f32_to_q7</a></div><div class="ttdeci">void aialgo_quantize_model_f32_to_q7(aimodel_t *model_f32, aimodel_t *model_q7, aitensor_t *representative_dataset)</div><div class="ttdoc">Quantize model parameters (weights and bias)</div></div>
</div><!-- fragment --><p>The Q7 model is now ready to use for example to <a class="el" href="_tutorial_inference_q7.html#RunInferenceQ7">run an inference</a>. You can also store the parameter memory buffer in your storage memory after the model is quantized and later on load it using the <a class="el" href="_tutorial_inference_q7.html#AutomaticLayerDeclaration">layer declaration described above</a>.</p>
<h1><a class="anchor" id="AutomaticQuantizationPython"></a>
Automatic quantization in Python</h1>
<p>To automatically quantize a model using python (for example a Keras or PyTorch model) you can use our AIfES Python tools. You can install the tools via pip with:</p>
<p><code>pip install <a href="https://github.com/Fraunhofer-IMS/AIfES_for_Arduino/raw/main/etc/python/aifes_tools.zip">https://github.com/Fraunhofer-IMS/AIfES_for_Arduino/raw/main/etc/python/aifes_tools.zip</a></code></p>
<p>The the quantized model can either be <a class="el" href="_tutorial_inference_q7.html#ManualLayerDeclaration">setup manually</a> with the calculated weights and quantization parameters (result_q_params, weights_q_params and weights_q7) or <a class="el" href="_tutorial_inference_q7.html#AutomaticLayerDeclaration">setup automatically</a> by using the buffer printed by the <code>print_flatbuffer_c_style()</code> function.</p>
<p><b>Example</b>: Quantize a <b>tf.keras</b> model:</p>
<div class="fragment"><div class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</div>
<div class="line"><span class="keyword">from</span> tensorflow <span class="keyword">import</span> keras</div>
<div class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</div>
<div class="line"><span class="keyword">from</span> tensorflow.keras <span class="keyword">import</span> layers</div>
<div class="line"><span class="keyword">from</span> aifes.tools <span class="keyword">import</span> quantize_model_q7, Layer, create_flatbuffer_q7, create_flatbuffer_f32, print_flatbuffer_c_style</div>
<div class="line"> </div>
<div class="line"><span class="comment"># ------------------------------------------------- Create and train the model in tf.keras --------------------------------------------------------</span></div>
<div class="line"> </div>
<div class="line">model = keras.Sequential()</div>
<div class="line">model.add(keras.Input(shape=(3,)))</div>
<div class="line">model.add(layers.Dense(3, activation=<span class="stringliteral">&quot;leaky_relu&quot;</span>))</div>
<div class="line">model.add(layers.Dense(2, activation=<span class="stringliteral">&quot;sigmoid&quot;</span>))</div>
<div class="line"> </div>
<div class="line">optimizer = keras.optimizers.Adam(lr=0.1)</div>
<div class="line">model.compile(optimizer=optimizer, loss=<span class="stringliteral">&quot;binary_crossentropy&quot;</span>)</div>
<div class="line"> </div>
<div class="line">model.summary()</div>
<div class="line"> </div>
<div class="line">X = np.array([[0., 0., 0.], </div>
<div class="line">              [1., 1., 1.], </div>
<div class="line">              [1., 0., 0.]])</div>
<div class="line"> </div>
<div class="line">T = np.array([[1., 0.], </div>
<div class="line">              [0., 1.], </div>
<div class="line">              [0., 0.]])</div>
<div class="line"> </div>
<div class="line">model.fit(X, T, batch_size=4, epochs=5)</div>
<div class="line"> </div>
<div class="line"><span class="stringliteral">&#39;&#39;&#39;</span></div>
<div class="line"><span class="stringliteral"># You may set the weights manually instead of training the model.</span></div>
<div class="line"><span class="stringliteral"></span> </div>
<div class="line"><span class="stringliteral">w1 = np.array([3.64540, -3.60981, 1.57631,</span></div>
<div class="line"><span class="stringliteral">            -2.98952, -1.91465, 3.06150,</span></div>
<div class="line"><span class="stringliteral">            -2.76578, -1.24335, 0.71257]).reshape(3, 3)</span></div>
<div class="line"><span class="stringliteral">            </span></div>
<div class="line"><span class="stringliteral">b1 = np.array([0.72655, 2.67281, -0.21291])</span></div>
<div class="line"><span class="stringliteral"></span> </div>
<div class="line"><span class="stringliteral">w2 = np.array([-1.09249, -2.44526,</span></div>
<div class="line"><span class="stringliteral">                 3.23528, -2.88023,</span></div>
<div class="line"><span class="stringliteral">                -2.51201,  2.52683]).reshape(3, 2)</span></div>
<div class="line"><span class="stringliteral"></span> </div>
<div class="line"><span class="stringliteral">b2 = np.array([0.14391, -1.34459])</span></div>
<div class="line"><span class="stringliteral"></span> </div>
<div class="line"><span class="stringliteral">weights = [w1, b1, w2, b2]</span></div>
<div class="line"><span class="stringliteral">model.set_weights(weights)</span></div>
<div class="line"><span class="stringliteral">&#39;&#39;&#39;</span></div>
<div class="line"> </div>
<div class="line"><span class="comment"># ------------------------------------------------- Convert and quantize the model to a flatbuffer --------------------------------------------------------</span></div>
<div class="line"> </div>
<div class="line"><span class="comment"># Representation of the model for AIfES pytools</span></div>
<div class="line">layers = [</div>
<div class="line">    Layer.DENSE_WT,     <span class="comment"># Dense / Fully connected layer with transposed weights (WT)</span></div>
<div class="line">    Layer.LEAKY_RELU,   <span class="comment"># Leaky ReLU layer. Add the alpha parameter to the act_params list</span></div>
<div class="line">    Layer.DENSE_WT,     <span class="comment"># Dense / Fully connected layer with transposed weights (WT)</span></div>
<div class="line">    Layer.SIGMOID       <span class="comment"># Sigmoid layer</span></div>
<div class="line">]</div>
<div class="line">act_params = [0.01]     <span class="comment"># Append additional parameters fore the activation functions here (e.g. alpha value for Leaky ReLU)</span></div>
<div class="line">weights = model.get_weights()</div>
<div class="line"> </div>
<div class="line"><span class="comment"># Platform specific settings</span></div>
<div class="line">ALIGNMENT = 4           <span class="comment"># For example ALIGNMENT = 4 on ARM Cortex-M or ESP32 controllers and ALIGNMENT = 2 on AVR controllers</span></div>
<div class="line">BYTEORDER = <span class="stringliteral">&#39;little&#39;</span>    <span class="comment"># &#39;little&#39; for little-endian or &#39;big&#39; for big-endian representation of the target system</span></div>
<div class="line"> </div>
<div class="line">result_q_params, weights_q_params, weights_q7 = quantize_model_q7(layers, weights, X, act_params=act_params)</div>
<div class="line"> </div>
<div class="line"><span class="comment"># Print required parameters for a Q7 model to console</span></div>
<div class="line">print()</div>
<div class="line">print(<span class="stringliteral">&quot;Layer result quantization parameters (shift, zero point):&quot;</span>)</div>
<div class="line">print(result_q_params)</div>
<div class="line">print(<span class="stringliteral">&quot;Weight and bias quantization parameters (shift, zero point):&quot;</span>)</div>
<div class="line">print(weights_q_params)</div>
<div class="line">print(<span class="stringliteral">&quot;Q7 weights:&quot;</span>)</div>
<div class="line">print(weights_q7)</div>
<div class="line">print()</div>
<div class="line"> </div>
<div class="line">flatbuffer_q7 = create_flatbuffer_q7(result_q_params, weights_q_params, weights_q7, target_alignment=ALIGNMENT, byteorder=BYTEORDER)</div>
<div class="line">flatbuffer_f32 = create_flatbuffer_f32(weights)</div>
<div class="line"> </div>
<div class="line"><span class="comment"># Print the parameter memory buffer to the console</span></div>
<div class="line">print(<span class="stringliteral">&quot;\nQ7:&quot;</span>)</div>
<div class="line">print_flatbuffer_c_style(flatbuffer_q7, elements_per_line=12, target_alignment=ALIGNMENT, byteorder=BYTEORDER, mutable=<span class="keyword">True</span>)</div>
<div class="line"> </div>
<div class="line">print(<span class="stringliteral">&quot;\nF32:&quot;</span>)</div>
<div class="line">print_flatbuffer_c_style(flatbuffer_f32, elements_per_line=8)</div>
</div><!-- fragment --><p><b>Example</b>: Quantize a <b>PyTorch</b> model</p>
<div class="fragment"><div class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</div>
<div class="line"><span class="keyword">import</span> torch</div>
<div class="line"><span class="keyword">from</span> torch <span class="keyword">import</span> nn</div>
<div class="line"><span class="keyword">from</span> aifes.tools <span class="keyword">import</span> quantize_model_q7, Layer, create_flatbuffer_q7, create_flatbuffer_f32, print_flatbuffer_c_style</div>
<div class="line"> </div>
<div class="line"><span class="comment"># ------------------------------------------------- Create and train the model in PyTorch --------------------------------------------------------</span></div>
<div class="line"> </div>
<div class="line"><span class="keyword">class </span>Net(nn.Module):</div>
<div class="line"> </div>
<div class="line">    <span class="keyword">def </span>__init__(self):</div>
<div class="line">        super(Net, self).__init__()</div>
<div class="line">        </div>
<div class="line">        self.dense_layer_1 = nn.Linear(3, 3)</div>
<div class="line">        self.leaky_relu_layer = nn.LeakyReLU(0.01)</div>
<div class="line">        self.dense_layer_2 = nn.Linear(3, 2)</div>
<div class="line">        self.sigmoid_layer = nn.Sigmoid()</div>
<div class="line"> </div>
<div class="line">    <span class="keyword">def </span>forward(self, x):</div>
<div class="line">        x = self.dense_layer_1(x)</div>
<div class="line">        x = self.leaky_relu_layer(x)</div>
<div class="line">        x = self.dense_layer_2(x)</div>
<div class="line">        x = self.sigmoid_layer(x)</div>
<div class="line">        <span class="keywordflow">return</span> x</div>
<div class="line">        </div>
<div class="line">        </div>
<div class="line">X = np.array([[0., 0., 0.], </div>
<div class="line">              [1., 1., 1.], </div>
<div class="line">              [1., 0., 0.]])</div>
<div class="line"> </div>
<div class="line">Y = np.array([[1., 0.], </div>
<div class="line">              [0., 1.], </div>
<div class="line">              [0., 0.]])</div>
<div class="line">              </div>
<div class="line">              </div>
<div class="line">X_tensor = torch.FloatTensor(X)</div>
<div class="line">Y_tensor = torch.FloatTensor(Y)</div>
<div class="line"> </div>
<div class="line">model = Net()</div>
<div class="line">criterion = nn.BCELoss()</div>
<div class="line">optimizer = torch.optim.Adam(model.parameters(), lr=0.1)</div>
<div class="line"> </div>
<div class="line">epochs = 200</div>
<div class="line">model.train()</div>
<div class="line"><span class="keywordflow">for</span> epoch <span class="keywordflow">in</span> range(epochs):</div>
<div class="line">    optimizer.zero_grad()</div>
<div class="line">    pred = model(X_tensor)</div>
<div class="line">    loss = criterion(pred, Y_tensor)</div>
<div class="line">    loss.backward()</div>
<div class="line">    optimizer.step()</div>
<div class="line"> </div>
<div class="line"><span class="comment"># ------------------------------------------------- Convert and quantize the model to a flatbuffer --------------------------------------------------------</span></div>
<div class="line"> </div>
<div class="line"><span class="comment"># Representation of the model for AIfES pytools</span></div>
<div class="line">layers = [</div>
<div class="line">    Layer.DENSE_WT,     <span class="comment"># Dense / Fully connected layer with transposed weights (WT)</span></div>
<div class="line">    Layer.LEAKY_RELU,   <span class="comment"># Leaky ReLU layer. Add the alpha parameter to the act_params list</span></div>
<div class="line">    Layer.DENSE_WT,     <span class="comment"># Dense / Fully connected layer with transposed weights (WT)</span></div>
<div class="line">    Layer.SIGMOID       <span class="comment"># Sigmoid layer</span></div>
<div class="line">]</div>
<div class="line">act_params = [0.01]     <span class="comment"># Append additional parameters fore the activation functions here (e.g. alpha value for Leaky ReLU)</span></div>
<div class="line">weights = [param.detach().numpy().T <span class="keywordflow">for</span> param <span class="keywordflow">in</span> model.parameters()]</div>
<div class="line"> </div>
<div class="line"><span class="comment"># Platform specific settings</span></div>
<div class="line">ALIGNMENT = 4           <span class="comment"># For example ALIGNMENT = 4 on ARM Cortex-M or ESP32 controllers and ALIGNMENT = 2 on AVR controllers (must equal AIFES_MEMORY_ALIGNMENT)</span></div>
<div class="line">BYTEORDER = <span class="stringliteral">&#39;little&#39;</span>    <span class="comment"># &#39;little&#39; for little-endian or &#39;big&#39; for big-endian representation of the target system</span></div>
<div class="line"> </div>
<div class="line">result_q_params, weights_q_params, weights_q7 = quantize_model_q7(layers, weights, X, act_params=act_params)</div>
<div class="line"> </div>
<div class="line"><span class="comment"># Print required parameters for a Q7 model to console</span></div>
<div class="line">print()</div>
<div class="line">print(<span class="stringliteral">&quot;Layer result quantization parameters (shift, zero point):&quot;</span>)</div>
<div class="line">print(result_q_params)</div>
<div class="line">print(<span class="stringliteral">&quot;Weight and bias quantization parameters (shift, zero point):&quot;</span>)</div>
<div class="line">print(weights_q_params)</div>
<div class="line">print(<span class="stringliteral">&quot;Q7 weights:&quot;</span>)</div>
<div class="line">print(weights_q7)</div>
<div class="line">print()</div>
<div class="line"> </div>
<div class="line">flatbuffer_q7 = create_flatbuffer_q7(result_q_params, weights_q_params, weights_q7, target_alignment=ALIGNMENT, byteorder=BYTEORDER)</div>
<div class="line">flatbuffer_f32 = create_flatbuffer_f32(weights)</div>
<div class="line"> </div>
<div class="line"><span class="comment"># Print the parameter memory buffer to the console</span></div>
<div class="line">print(<span class="stringliteral">&quot;\nQ7:&quot;</span>)</div>
<div class="line">print_flatbuffer_c_style(flatbuffer_q7, elements_per_line=10, target_alignment=ALIGNMENT, byteorder=BYTEORDER, mutable=<span class="keyword">True</span>)</div>
<div class="line"> </div>
<div class="line">print(<span class="stringliteral">&quot;\nF32:&quot;</span>)</div>
<div class="line">print_flatbuffer_c_style(flatbuffer_f32, elements_per_line=8)</div>
</div><!-- fragment --> </div></div><!-- contents -->
</div><!-- PageDoc -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
  <ul>
    <li class="footer">Generated by <a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.9.1 </li>
  </ul>
</div>
</body>
</html>
